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Abstract: With technological advances, it is now possible to use games to capture information-rich behaviours that reveal 
processes by which players interact and solve problems. Recent problem-based games have been designed to assess and 
record detailed interactions between the problem solver and the game environment, and thereby capture salient solution 
processes in an unobtrusive way (Zoanetti, 2010; Bennett et al., 2003; Shute & Wang, 2009). The Assessment and Teaching 
of 21st Century Skills project used an innovative approach to capture these processes through responses (activities and 
communication patterns) within collaborative games (Griffin & Care, 2015). Game player response patterns are identified 
as behaviours that are indicative of skills and are captured in real-time game play within time-structured log files. The 
analysis of these log files allows for inferences to be drawn in regard to the efficiency and quality of player performance. A 
concern with this approach is that game development for this purpose is complex, time consuming and expensive, with 
unique scoring applied to each game. This paper presents another approach that identifies, across games, common 
behaviours. A systematic scoring system for assessing player behaviours in games could provide access to useful data not 
only from new games but from existing games. This paper presents such an approach using collaborative games situated in 
a problem-solving context. 
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1. Introduction 

It is unsurprising that references to 21st century skills have started many debates over what and how today's 
students should learn in order to become productive citizens (Kim & Shute, 2015). Lists of 21 st century skills 
typically include problem solving, critical thinking, decision making and collaboration (Binkley et al., 2012). 
While these skills are not new, their relevance is greater than ever in this information age, where dissection of 
information, rather than just consumption of information, is critical. The increasingly rapid spread of 
sophisticated technologies into every facet of society is causing significant shifts, including how educational 
systems should be structured to prepare students effectively for life and work in the 21st century. Knowledge 
workers, now confronted by complex problems, need to be able to think systemically, creatively and critically, 
and at the same time be able to apply extensive problem solving and complex communication (McClarty et al., 
2012) skills. Continuing to provide the same types of education to students as the world continues to change 
will not serve them well. The skills and knowledge exposed by traditional education are no longer seen as 
adequate preparation for success in life. Moreover, traditional assessments often fail to assess complex skills 
or measure what students can do with the knowledge and skills acquired through education (Shute & Kim, 
2014). 

2. Embedding assessments in games 

How and what we learn is rapidly changing and researchers argue that games possess unique affordances to 
address complex problem-solving skill development that our current educational system is failing to provide 
(OECD, 2013). Assessment generally refers to the process of gathering information about competencies and 
attributes (Kim & Shute, 2015). Games can be powerful vehicles to assess competencies and dispositions, such 
as persistence, flexibility, creativity, self-efficacy, critical thinking, systems thinking, openness, problem-solving 
and teamwork, which can positively impact student academic achievement and other aspects of life (Delacruz, 
2011; Zein & Diab, 2015; Care et al., 2015; Wang et al., 2015; Deniz et al., 2014; Kim & Shute, 2015)). 
Technologies, as well as educational and psychological measurement approaches, have advanced in recent 
times (Wang, Shute, & Moore, 2015). Technologies allow us to embed assessments into games by extracting 
multifaceted evidence from a learner to infer competency levels across a network of skills, addressing what 
the person knows and can demonstrate (Shute & Kim, 2014). 


ISSN 1479-4403 335 ©ACPIL 

Reference this paper as: Scoular C, Care E and Awwal N, "An Approach to Scoring Collaboration in Online Game 
Environments" The Electronic Journal of e-Learning Volume 15 Issue 4 2017, (pp335-342) available online at www.eiel.org 







The Electronic Journal of e-Learning Volume 15 Issue 4 2017 


One area of promise is a movement toward the use of well-designed games to provide meaningful student 
assessment environments that require the application of various complex competencies. Traditional 
assessments, such as paper and pencil multiple choice tests, do not necessarily capture the dynamic and 
complex nature of 21 st century skills. Many researchers are exploring the possibility that games are well suited 
to improve instruction and differentiate learning while also providing more effective and less intrusive 
measurement than traditional assessments can offer (McClarty et al., 2012). Gee (2007) argues that if games 
are designed appropriately, they will stimulate higher-order thinking and collaborative activities among 
participants. Games are likely to address 21 st century skill development through active interactions, 
collaborative experiences and complex problem solving (Schifter, 2013). Thus, embedding an approach to 
scoring within games provides a way to monitor participants' level of competency and provide useful feedback 
(Shute & Kim, 2014). Assessment is a process of drawing reasonable inferences about what a person knows by 
evaluating what they do, say, make or write in a given situation (Griffin & Care, 2015). Games can be designed 
to increase in complexity and cognitive demand as play progresses, such that gameplay can be used for 
assessment purposes (Delacruz, 2011; Gee, 2007). 

The use of games and simulations in assessment and learning is expected to increase over the next few years 
(McClarty et al., 2012). Additionally, assessments in this format have the advantage of providing teachers with 
tools that help them tailor instruction to the needs of students at different achievement levels (Mayrath, 
Clarke-Midura, Robinson, & Schraw, 2012). Test takers' skills in these virtual environments can be 
automatically assessed by tracking their behaviours. Moreover, analysis of log files can highlight the strengths 
and weaknesses of game design (Ifenthaler et al., 2014; O'Neil, Chuang, & Chung, 2004). 

The use of games in education so far reports mixed results, despite theoretical endorsement of its benefits. 
Design, development and implementation of games is expensive, time-consuming, resource-demanding and 
challenging (Awwal, Alom, & Care, 2015), and the goals of games do not necessarily align with learning or 
assessment goals: ostentatious game features can distract players from their task. A scientific attitude with 
regard to the design of educational games requires validated measures of learning outcomes and associated 
assessment methods in order to determine which design elements work best, when, and why (Ifenthaler et al., 
2014). Scholars from various disciplines have found that well-designed games require problem solving skills, 
persistence, and creativity (Gee, 2007; Prensky, 2004; Shaffer, Squire, Halverson, & Gee, 2005); these are 
competencies critical to success in the 21st century, but to date are not explicitly supported by education 
systems. With appropriate design, student learning of skills through games can be promoted (Moreno-Ger, 
Burgos, Martinez-Ortiz, Sierra, & Fernandez-Manjon, 2008). 

3. Measuring collaboration 

Theorists have for decades emphasised the importance of social interaction during learning (e.g. Vygotsky, 
1978; Dillenbourg, 1999). From a learning perspective, the benefits of working collaboratively lie in the process 
rather than the end goal. If an individual is faced with a problem, they may have ways and means to solve it 
alone. However, if a student collaborates to solve a problem they enter into an iterative process in which they 
engage with others, understand perspectives, and regulate social interaction. A collaborative approach 
provides different learning opportunities than working individually. However, it is reasonable to presume that 
it is in the individual's interests to learn how to work alone as well as to collaborate. 

Game theory has been based in two long-standing categories of people working together: competitive and 
cooperative. Marschak and Radner (1972) recognised that a team is an organisation in which players can 
possess different information and skills, and individuals need to work together to maximise a joint outcome. 
Recent research places competition, cooperation and collaboration on a spectrum with competition and 
collaboration at opposite ends, determined by whether players have a shared goal (Zagal, Rick, & Hsi, 2006). In 
competitive games, participants typically work against one another, whereas in cooperative games they work 
together. The key difference in collaborative games is that there is a necessity for players to work together, 
with each player contributing uniquely to the game. Collaborative games can be designed to reduce any 
competitiveness players may initially possess. This can be addressed by providing shared goals and ensuring 
dependency between players through division of information and resources. Such strategies may constitute 
effective learning mechanisms. Cooperation, collaboration, and competition are combined in real life, where 
workers at times understand that they need to work together in order to achieve their goals, notwithstanding 
that they may also be competitors. 
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Critical to the integration of collaboration into current education systems is development of tools and 
strategies for assessing the skill. There is recognition that collaborative assessment requires human-human 
interactions, complex problems that cannot be solved by a single individual, and diverse sets of knowledge, 
skills and experience (Hesse et al., 2015). Assessing individuals' collaboration while also ensuring a reliable, 
valid, and fair assessment of their skills represents a major advance in educational measurement (von Davier & 
Halpin, 2013). Measuring less explored skills such as collaboration therefore requires methods of assessment 
design and psychometric analysis that may not be contained within traditional methods of assessment. In 
order to assess 21 st century skills such as collaboration, inferences about student ability are required, and 
technology is providing ways to capture the relevant data from which these inferences may be drawn. Log 
stream data are a familiar feature of computer use, but have only recently been generated at scale to provide 
new insights and perspectives in education. Log stream data capture student activities and retain a time- 
stamped record as students work through an online game. Students can thus be scored in real time in an 
automated approach that reduces the time and effort required for teachers to observe student performance in 
the classroom. 

4. Method 

4.1 Design 

The online games described harness log stream technology transforming them into assessments of 
collaboration. Behaviours such as action and chat communication are captured in real-time in a time-series log 
file, allowing for reasonable interpretations of such behaviours to be made in relation to a player's 
collaborative ability. The games presented act as a neat illustration of the similarity of the skills applied across 
collaborative games and therefore of the opportunity to capture similar behaviours. The method outlined was 
designed for a pilot study to test an approach to scoring involving common behaviours across games. The 
analysis presented in this paper involved three collaborative games developed by the ATC21S project: 
Laughing Clowns, Balance Beam, and Game of 20. 

4.2 Participants 

Each game was administered to 410 students (186 males and 224 females) across six schools in Victoria, 
Australia. The principal in each school was responsible for selecting which classes and, therefore, which 
students were involved in the trial. The principal selected classes primarily on the basis of availability for 
timetabled visits. The students were 12 to 15 years old (X = 14 years). 

4.3 Materials 

The games presented in this study were designed for two players to work together in an online environment. 
The players are provided a login and required to work through the assessment together. Their communication 
is facilitated by an embedded chat box. Players can collaborate on the game irrespective of physical location, 
provided they login simultaneously. Players cannot progress through the assessment independent of one 
another, and often cannot solve the problem without input from their partner. From the players' perspective, 
these assessments present as games. The log stream data generated allows for identification and scoring of 
player behaviours during game play. 

In the Laughing Clowns game, illustrated in Figure 1, the players are presented with a clown machine each and 
12 balls to be shared between them. The goal for the players is to determine whether the two clown machines 
work in the same way. In order to do this, the two players need to share information that is available to each 
of them and discuss the rules, as well as negotiate how many balls they should each use for problem 
resolution. This game requires players to find patterns, share resources, form rules and reach conclusions 
(Care et al., 2015). 
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Finished | 

You and your partner both have a Laughing Clown machine in front of you. Drop 
balls into the clown's mouth to see how yours works. Find out if your machine 
works exactly the same as your partner's. You have 12 balls to share. 

You and your partner both have a Laughing Clown machine in front of you. Drop 
balls into the clown's mouth to see how yours works. Find out if your machine 
works exactly the same as your partner's. You have 12 balls to share. 

Chat display 

Chat display 

unit iHfcs 


““ 

Our the same 


.. Our machines the same 

Our machines 


CH* input send message 

TVT 


Chat input send message 

Student A View 

Student B View 


Figure 1: Screenshots of the Laughing Clowns game. 


The Balance Beam game, illustrated in Figure 2 , is presented so that each player can interact with only one side 
of the beam and is unable to see their partner's side. Both players are then provided with a limited number of 
weights. To make the process more collaborative, one player begins with all the weights, hence requiring 
sharing across players to progress through the task. Initially, weights must be passed from one player to the 
other; then each must place their weights in the correct position on the balance beam in order to achieve 
balance. Players need to apply and test rules in order to balance the beam, possibly leading to multiple correct 
solutions. 



1. You and your partner share this set of scales. Explore what happens when 
you use different masses. Can you make the scales balance? 



1. You and your partner share this set of scales. Explore what happens when 
you use different masses. Can you make the scales balance? 


You can send your partner different masses, one at a time, by dragging 
them onto the transfer pad. 

The scales are balanced 

HZ 

We are ready to move on ^ 


SZ3 

X 


Student A View 


Your partner can send you a mass. You can send it back by dragging it 
onto the transfer pad. 

The scales are balanced 

We are ready to move on | 


X 


Student B View 


Figure 2: Screenshots of the Balance Beam game 


The Game of 20, illustrated in Figure 3, presents a mathematics based problem, in which the object is to work 
out the ideal numbers to reach a sum of 20 before the competitor, which in this game is the computer agent, 
reaches that sum. The two players must collaborate in order to win against the computer. Players need to take 
turns at selecting a number, with their aggregated number the one ultimately played. Instructions provided to 
the players include "Where will your team place the counter to be sure of winning the game?" and 
"Remember that the number of counter advances will be a combination of your number and your partner's 
number". These instructions make clear both the competitive and collaborative nature of the game - that 
together the players need to win against a third party - the computer. 
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In this task you and your partner are going to play a game as a 
team aganst the computer 


The object of the game is to make a total of exactly 20 


Your team and the computer take it in turns to cover up a 
number on the board wth a counter 


Your team needs to work together to move the number of 
spaces you want each turn 


To get the idea, play the game as many ernes as you eke 
against the computer 




YOU*? NUMBER 


PARTNER NUMBER Q 

max = 5 


mrmTTrrii 


In this task you and your partner are going to play a game as a 
team against the computer 


The object of the game is to make a total of exactly 20 


Your team and the computer take it in turns to cover up a 
number on the board with a counter 


Your team needs to work together to move the number of 
spaces you want each tun 


To get the idea, play the game as many times as you fcke 
against the computer 


PARTNER NUMBER 


YOUR NUMBER 


Student A View 


Student B View 


Figure 3: Screenshots of the Game of 20. 

4.4 Procedure 


This paper outlines four behaviours identified as common across players and across each of the games. Each 
player completed all three games with the same partner within a classroom environment, taking 
approximately one hour. Log stream data was used to identify actions and communications undertaken and 
interactions between players. These observable behaviours are identified from the log stream data by 
employing search algorithms. The algorithms search the stream for the behaviours and score each behaviour 
as either present or absent for each player in each game. All activities and interactions that are possible within 
a game environment, if recorded systematically (records of player-game interactions), can measure salient 
solution processes in an unobtrusive way (Bennett et al., 2003; Zoanetti, 2010). These recorded detailed 
interactions between the problem solver and the game environment can be linked to each player's level of 
proficiency and used to evaluate the process and efficiency with which problem solvers complete games 
(Pellegrino, Chudowsky, & Glaser, 2001; Williamson, Mislevy, & Bejar, 2006). 

5. Results 


The pilot study identified four common behaviours across the three games (Table 1). The percentages in the 
columns on the right refer to the frequency of each behaviour in each game, measured across all participating 
players. 

Table 1: Common behaviours 


Number 

Behaviours 

Scoring 

Clowns 

Balance 

Game of 




(%) 

(%) 

20 (%) 

1 

Sends partner resource 

0 (absent); 1 (present) 

54 

44 

57 

2 

Presence of chat immediately before selecting 
final answer 

0 (absent); 1 (present) 

34 

23 

32 

3 

Trials all resources 

0 (absent); 1 (present) 

34 

55 

62 

4 

Asks partner a question 

0 (absent); 1 (present) 

75 

71 

74 


Behaviour number one identifies the percentage of players sending their partner a resource. This is an 
important component of collaboration, since to work together players need to identify and share each other's 
resources. Further, the asymmetrical nature of the games makes this sharing critical for solving the problem. 
Figure 4 provides an example of the log stream data from a pair of players in the Balance Beam game. This 
excerpt shows the data used to identify behaviour number one. The identification of the 'pass resource' data 
event, as indicated by the green arrow, represents the passing of a resource from one player to another. In this 
example, it is the passing of the 300g mass from player A to player B. To record the presence of this behaviour, 
the search algorithm for behaviour number one searched the log stream data for the presence of a 'pass 
resource' data event. If present, a value of 1 was recorded for that player against that behaviour. If absent, a 0 
value was recorded. Instances of sending resources were identified in all three games. 
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team id 

student id 

task id 

role 

page 

event 

argl 

arg2 

arg3 

outcome 

timestamp 

usapri0021 

usapri0021b 

BALANCE 

B 

3 

chat 

same again 1 have no weights 




1/10/2015 11:42 

usapri0021 

usapri0021a 

BALANCE 

A 

3 

chat 

yeh 1 have 4 again 




1/10/2015 11:42 

usapri0021 

usapri0021b 

BALANCE 

B 

3 

chat 

can you give me 300g? 




1/10/2015 11:42 

usapri0021 

usapri0021b 

BALANCE 

B 

3 

chat 

ill try it on 1 like last time 




1/10/2015 11:42 

usapri0021 

usapri0021a 

BALANCE 

A 

3 

pass-resource 

BOOg 

B M 



1/10/2015 11:43 

usapri0021 

usapri0021a 

BALANCE 

A 

3 

chat 

kdone 




1/10/2015 11:43 

usapri0021 

usapri0021b 

BALANCE 

B 

3 

move-resourc 

300g 

1 



1/10/2015 11:43 

usapri0021 

usapri0021a 

BALANCE 

A 

3 

chat 

k ill put lOOg on 3 




1/10/2015 11:44 

usapri0021 

usapri0021a 

BALANCE 

A 

3 

move-resourc 

100g 

3 



1/10/2015 11:44 


Figure 4: Example of log stream data from the Balance Beam game showing the 'pass resource' data event. 

Table 1 presents the percentage of players for whom the presence of each behaviour was recorded in each 
game. Recording a present behaviour, resulting in a '1' being recorded for that player in that game on that 
specific behaviour, is much like being scored correctly on a traditional test: for the purpose of assessment, 
behaviours are like test items. The frequencies of percentages correct (or, in this case, presence of a 
behaviourr) can be taken as a proxy measure of the perceived difficulty of that behaviour (Adams et al., 2015). 
If a large proportion of players demonstrate a particular behaviour, this suggests that behaviour was easier to 
complete than those behaviours demonstrated by only a small proportion of the players. In the case of 
behaviour number one, the frequency appears reasonably consistently across the three games, at a mid-range 
(between 44-57%). 

Behaviour number two identifies the presence or absence of chat immediately before the final answer in the 
game is selected. Discussion of suggestions or answers to the problem before entering them suggests good 
collaboration. From a game design perspective, there needs to be an opportunity to select a final answer to 
make this behaviour viable. As can be seen in Table 1, there is a relatively low frequency of players 
demonstrating this behaviour and this is consistent across games (between 23-34%). 

Behaviour number three identifies whether players engage with every feature of the game. In problem solving, 
a systematic approach requires trialling, and in the collaborative context, this is made more complex by virtue 
of different resources being available to each player. In games of this kind, players benefit from pooling and 
trialling all joint resources. In the games demonstrated, some resources are redundant or of little use to the 
current activity, and players need to trial resources to assist in identifying the useful ones. Behaviour number 
three increases in frequency across the games suggesting that more players engaged with all of the features in 
the last game. Game of 20 (62%), than they did in the earlier game. Laughing Clowns (32%). It is possible 
players become more familiar with the interactive elements of the design in later games, or they become more 
confident in engaging with the game features. 

Behaviour number four identifies whether a player asks their partner a question, indicated by the presence of 
key question words such as 'what', 'why' or 'how' in the chat box. Seeking more information from others is 
critical to building a shared understanding of the problem and will provide a foundation from which further 
collaboration can be built. The frequencies for this behaviour were consistently high across the three games 
(between 71-75%) suggesting that most players recognised the benefits of potential contributions from their 
partner. 

6. Discussion 

The approach taken in this paper highlights the ability to measure collaboration through games. In addition, it 
demonstrates that common behaviours can be identified across games to make inferences about player 
collaborative ability. Although the three games presented in this study demonstrated different problems and 
different requirements for collaboration, behaviours common to all three could still be identified. The 
identification of common behaviours makes scoring much more efficient, eliminating the laboriousness of 
scoring each game separately. Not only does the log stream data provide insight into what players are learning 
within the games, it provides an approach for assessing that learning. 

Advances in technology have expanded the availability and opportunities for assessment game design. 
Computer-based performance assessment allows information to be captured, in relation to the processes and 
responses of the player (Bennett, Persky, Weiss, & Jenkins, 2007), that paper-based assessments cannot 
provide. This development also clearly possesses administrative advantages, such as automated scoring, 
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greater reach, instant feedback and a reduction in subjective scoring. The approach presented here allows for 
games to be transformed into assessments without the player experience or interface changing. This presents 
an engaging situation for students in the classroom and an opportunity to generate data for teachers to 
evaluate their students' collaboration skills. Future work will focus on identifying additional common 
behaviours in games from which the same inferences might reasonably be drawn. 

7. Conclusion 

The approach outlined represents an innovation in assessment, moving from a model in which abilities are 
identified by direct responses to questions with correct and incorrect answers to a behavioural inference 
model. In this model, inferences about capabilities are made on the basis of individuals' actions and 
communications in a game environment. The challenge for educators who want to employ games to support 
learning is to make valid inferences about what the students know and can do without disrupting their 
engagement with the game (Shute & Kim, 2014; Wang et al., 2015). The use of capture of actions through log 
streams provides this facility without students needing to consciously respond to assessment prompts. 
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